Skip to content

Fixes #26774: expose custom properties in SpEL policy rule conditions#27218

Open
mohitjeswani01 wants to merge 8 commits intoopen-metadata:mainfrom
mohitjeswani01:feat/26774-custom-properties-spel-policy-conditions
Open

Fixes #26774: expose custom properties in SpEL policy rule conditions#27218
mohitjeswani01 wants to merge 8 commits intoopen-metadata:mainfrom
mohitjeswani01:feat/26774-custom-properties-spel-policy-conditions

Conversation

@mohitjeswani01
Copy link
Copy Markdown

@mohitjeswani01 mohitjeswani01 commented Apr 9, 2026

Describe your changes:

Fixes #26774

I worked on the Policy engine's SpEL evaluation context because entity
Custom Properties (e.g., "Data Sensitivity", "Department Owner") were
completely invisible to Policy Rule conditions — forcing teams to use
overly broad roles or Tags as a workaround instead of proper
attribute-based access control (ABAC).

Root Cause

ResourceContext never fetched the extension field (where custom
properties are stored) when resolving entities for policy evaluation.
Even if extension was available, no SpEL-callable method exposed it
to rule conditions.

What I Changed — 3 files, 1 commit:

ResourceContextInterface.java

  • Added getCustomProperties() default method returning
    Map<String, Object> — safe default of empty map

ResourceContext.java

  • Implemented getCustomProperties() — fetches entity extension,
    deserializes via JsonUtils, returns empty map on any failure
  • Extended resolveEntity() field list to include FIELD_EXTENSION
    when the entity type supports it — so custom properties are
    actually loaded from DB during policy evaluation

RuleEvaluator.java

  • Added getCustomProperties() bridge method — returns emptyMap()
    (not null) when resourceContext is null, preventing NPE in SpEL
  • Added matchCustomProperty(String propertyName, String expectedValue)
    — a convenient boolean helper for policy rule conditions

Usage in Policy Rule Conditions

After this change, policy rules can use:

// Direct map access
resource.customProperties.get('dataSensitivity') == 'PII'

// Convenient helper method (new)
matchCustomProperty('dataSensitivity', 'PII')
matchCustomProperty('department', 'Finance')

Why matchCustomProperty() matters

The issue explicitly requested a helper method to simplify syntax.
Without it, policy authors must use verbose map access syntax.
matchCustomProperty() handles all null/missing cases gracefully
and works correctly in both validation mode and runtime mode.

How I Tested

Added 6 unit tests to RuleEvaluatorTest.java covering:

  • getCustomProperties() returns empty map when no extension
  • getCustomProperties() returns correct map when extension is set
  • matchCustomProperty() returns true when property matches
  • matchCustomProperty() returns false when value doesn't match
  • matchCustomProperty() returns false when property is missing
  • matchCustomProperty() returns false when entity has no extension

Type of change:

  • New feature

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes #26774: expose custom properties in SpEL policy rule conditions
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
  • The issue properly describes why the new feature is needed, what's the goal, and how we are building it.
  • I have added tests around the new logic.

Summary by Gitar

  • Refactored caching infrastructure:
    • Implemented single-flight loading and thread-safe cache invalidation for EntityRepository read bundles
    • Added robust cache-eviction patterns for rename-cascade flows and bulk updates
  • Improved version history management:
    • Introduced schema-aware version field tracking with fallback support for legacy records
    • Added optimized batch insertion logic for entity version extensions
  • Optimized read performance:
    • Implemented fast-path certification checks and added container relationship caching to minimize DB round-trips

This will update automatically on new commits.

Copilot AI review requested due to automatic review settings April 9, 2026 21:41
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 9, 2026

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the policy engine’s SpEL evaluation capabilities so policy conditions can evaluate entity Custom Properties (stored in the entity extension field), enabling ABAC-style rules based on those properties.

Changes:

  • Add getCustomProperties() to ResourceContextInterface and implement it in ResourceContext by deserializing the entity extension.
  • Update ResourceContext.resolveEntity() to include extension in the resolved field list when supported, so custom properties are available during policy evaluation.
  • Add getCustomProperties() and matchCustomProperty(...) helpers to RuleEvaluator, plus unit tests covering the new behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File Description
openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/ResourceContextInterface.java Adds a default getCustomProperties() contract returning an empty map.
openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/ResourceContext.java Implements custom property extraction from extension and attempts to ensure extension is fetched during entity resolution.
openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/RuleEvaluator.java Adds SpEL-callable accessors/helpers for custom properties.
openmetadata-service/src/test/java/org/openmetadata/service/security/policyevaluator/RuleEvaluatorTest.java Adds unit tests for custom property access and matching behavior.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 9, 2026

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@mohitjeswani01
Copy link
Copy Markdown
Author

@harshach @pmbrull all review comments have been addressed:

✅ Gitar Bot — 2/2 resolved (Quality + Performance)
✅ Copilot — @function annotations added, test cleanup done,
line length fixed, caching implemented

Could you please add the safe to test label to trigger CI? thank you!🙏

@PubChimps PubChimps added the safe to test Add this label to run secure Github workflows on PRs label Apr 16, 2026
@github-actions
Copy link
Copy Markdown
Contributor

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@mohitjeswani01 mohitjeswani01 force-pushed the feat/26774-custom-properties-spel-policy-conditions branch from e51d312 to 2e365ad Compare April 16, 2026 02:19
Copilot AI review requested due to automatic review settings April 16, 2026 02:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

@mohitjeswani01
Copy link
Copy Markdown
Author

Hi @PubChimps 👋

I've just applied the spotless formatting fix — Java checkstyle should
now pass and also fixed the recent Bots comments

I noticed the related PR #27033 was closed and Issue #26774 was closed
as completed with my PR still open. My PR #27218 implements the full
feature:

  • Exposes custom properties in the SpEL evaluation context via
    ResourceContext
  • Adds matchCustomProperty() helper method for simplified syntax

Could you let me know if this PR is still being considered ,
or if there are specific changes needed? I'm actively monitoring CI
and will address any feedback immediately.

Thank you! 🙏

@PubChimps
Copy link
Copy Markdown
Contributor

Hi @mohitjeswani01 thanks for reaching out, this issue was closed to prevent further PR's from being opened on it, could you please take a lot a the failed checks that were run?

@mohitjeswani01
Copy link
Copy Markdown
Author

Hi @mohitjeswani01 thanks for reaching out, this issue was closed to prevent further PR's from being opened on it, could you please take a lot a the failed checks that were run?

yes @PubChimps thanks for informing! I will make sure to update you shortly on the failed check , thanks !

Copilot AI review requested due to automatic review settings April 22, 2026 13:56
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 22, 2026

🟡 Playwright Results — all passed (12 flaky)

✅ 3970 passed · ❌ 0 failed · 🟡 12 flaky · ⏭️ 86 skipped

Shard Passed Failed Flaky Skipped
🟡 Shard 1 298 0 1 4
🟡 Shard 2 740 0 5 8
🟡 Shard 3 753 0 2 7
🟡 Shard 4 758 0 1 18
🟡 Shard 5 686 0 1 41
🟡 Shard 6 735 0 2 8
🟡 12 flaky test(s) (passed on retry)
  • Pages/AuditLogs.spec.ts › should apply both User and EntityType filters simultaneously (shard 1, 1 retry)
  • Features/ActivityAPI.spec.ts › Activity event is created when description is updated (shard 2, 1 retry)
  • Features/ActivityAPI.spec.ts › Activity event shows the actor who made the change (shard 2, 1 retry)
  • Features/AdvancedSearchSuggestions.spec.ts › Verify suggestions for Tier field (shard 2, 1 retry)
  • Features/BulkImportWithDotInName.spec.ts › Full import cycle with dot in service name (shard 2, 1 retry)
  • Features/IncidentManager.spec.ts › Complete Incident lifecycle with table owner (shard 2, 1 retry)
  • Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
  • Flow/NotificationAlerts.spec.ts › Conversation source alert (shard 3, 1 retry)
  • Pages/DataContracts.spec.ts › Create Data Contract and validate for Store Procedure (shard 4, 1 retry)
  • Pages/ExplorePageRightPanel.spec.ts › Should verify deleted user not visible in owner selection for table (shard 5, 1 retry)
  • Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
  • Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@mohitjeswani01
Copy link
Copy Markdown
Author

mohitjeswani01 commented Apr 22, 2026

Hii @PubChimps sir , thanks for informing about the CI the Ci's are passing now and i will actively monitor those failing checks also and address the copilot comments shortly 👍!

@gitar-bot
Copy link
Copy Markdown

gitar-bot Bot commented Apr 29, 2026

Code Review ⚠️ Changes requested 4 resolved / 8 findings

Exposes custom properties in SpEL policy rule conditions while resolving deserialization and caching issues. Refactoring is required to address silent failures in seed data initialization, excessive method complexity in fillReadBundle, and inefficient list streaming.

⚠️ Edge Case: initSeedDataFromResources now silently continues on failure

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1231-1241

The try/catch added around initializeEntity(entity) means that if a critical seed entity (e.g. a default policy, role, or bot) fails to initialize, the system will start up with missing seed data rather than failing fast. The previous behavior propagated the exception, halting seed loading and surfacing the problem immediately.

While resilience against one bad seed entity is useful, silently continuing could leave the system in a subtly broken state (e.g. missing the ingestion-bot or a default admin role). Consider distinguishing between optional and required seed entities, or at minimum collecting all failures and throwing at the end if any occurred.

Suggested fix
// After the loop, fail if any critical seed entity was skipped:
List<String> failures = new ArrayList<>();
for (T entity : entities) {
  try {
    initializeEntity(entity);
  } catch (Exception e) {
    LOG.warn("Failed to init {} '{}'", entityType, entity.getFullyQualifiedName(), e);
    failures.add(entity.getFullyQualifiedName());
  }
}
if (!failures.isEmpty()) {
  throw new IOException("Failed to initialize seed entities: " + failures);
}
💡 Quality: fillReadBundle has 6 params and 3 boolean tracking vars

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1655-1669

The fillReadBundle method (lines 1655-1813) is ~160 lines long with 5 parameters, 3 boolean state-tracking variables, and multiple nested branches. Per the project's complexity guidelines (max 15 lines/method, max 5 params), this method would benefit from being decomposed. For example, the cache-hit path (lines 1665-1684), the DB-fetch path (lines 1689-1724), and the cache-populate path (lines 1756-1765) are each self-contained and could be extracted.

💡 Performance: insertVersionExtensionsWithMetadata streams list 6 times

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:2597-2611

In insertVersionExtensionsWithMetadata (lines 2597-2618), the versionExtensionRecords list is streamed 6 separate times to extract each field into its own list. For large batch imports this is wasteful. A single loop collecting all fields simultaneously would be more efficient and more readable.

Suggested fix
List<UUID> ids = new ArrayList<>(versionExtensionRecords.size());
List<String> exts = new ArrayList<>(versionExtensionRecords.size());
// ... etc for each field
for (var r : versionExtensionRecords) {
  ids.add(r.getId());
  exts.add(r.getExtension());
  jsons.add(r.getJson());
  versions.add(r.getVersionNum());
  keys.add(r.getChangedFieldKeys());
}
daoCollection.entityExtensionDAO()
  .insertVersionExtensions(ids, exts, entityType, jsons, versions, keys);
💡 Quality: extractFqn line exceeds typical length limit

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:3262

Line 3262 (return node.hasNonNull("fullyQualifiedName") ? node.get("fullyQualifiedName").asText() : null;) contains the magic string "fullyQualifiedName" repeated twice. Per the 'no magic strings' guideline, this should use the existing FIELD_FULLY_QUALIFIED_NAME constant (or equivalent) from the Entity class.

✅ 4 resolved
Quality: Use @Getter for supportsExtension instead of try-catch workaround

📄 openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/ResourceContext.java:235-242
The try-catch block in resolveEntity() to check extension support (lines 235-242) catches all Exception types, which could silently swallow unrelated errors (e.g., database connectivity issues, OOM). Other field support checks use dedicated boolean getters like isSupportsOwners(), isSupportsTags(), etc. The supportsExtension field in EntityRepository already exists but lacks a @Getter annotation, unlike the other support flags.

This broad catch clause could mask real failures during policy evaluation, making debugging harder.

Performance: getCustomProperties() re-deserializes extension on every call

📄 openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/ResourceContext.java:192-206
ResourceContext.getCustomProperties() calls JsonUtils.convertValue(entity.getExtension(), Map.class) every time it's invoked, without caching the result. If a policy has multiple custom property conditions (e.g., matchCustomProperty('sensitivity', 'PII') && matchCustomProperty('dept', 'Finance')), the extension is deserialized repeatedly for the same entity in the same evaluation.

While not critical for typical workloads, caching the deserialized map would be trivial and avoids redundant work, especially if custom property conditions become popular in policy rules.

Bug: Stale cachedCustomProperties causes test failures across tests

📄 openmetadata-service/src/test/java/org/openmetadata/service/security/policyevaluator/RuleEvaluatorTest.java:1148-1154 📄 openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/ResourceContext.java:43 📄 openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/ResourceContext.java:194-208
The cachedCustomProperties field on ResourceContext is never cleared between tests. The test class creates a single resourceContext in @BeforeAll (line 224) and resetContext() in @AfterEach only recreates the RuleEvaluator/EvaluationContext — it does not recreate resourceContext.

Once any test calls getCustomProperties() (directly or via matchCustomProperty), the result is cached in cachedCustomProperties. Subsequent tests that change table.setExtension(...) will still get the stale cached value.

For example, if test_getCustomProperties_noExtension runs first (caches emptyMap), then test_matchCustomProperty_matches will fail because the cache returns an empty map despite the test setting extension to {dataSensitivity: PII}.

This is an order-dependent test failure — it may or may not manifest depending on JUnit 5's method ordering.

Edge Case: matchCustomProperty uses toString() — may mismatch for non-string types

📄 openmetadata-service/src/main/java/org/openmetadata/service/security/policyevaluator/RuleEvaluator.java:456-460
Custom properties can be non-string types (integers, booleans, dates, arrays, objects). matchCustomProperty compares via expectedValue.equals(actual.toString()). For example, a boolean custom property stored as true (Java Boolean) would have toString() = "true", which works. But a list/array like ["a","b"] would produce an unhelpful toString() representation, and an integer 42 would need to be matched as matchCustomProperty('count', '42') — which is workable but not obvious.

Consider documenting that matchCustomProperty is designed for string/simple-valued custom properties, or handling richer types (e.g., list contains check).

🤖 Prompt for agents
Code Review: Exposes custom properties in SpEL policy rule conditions while resolving deserialization and caching issues. Refactoring is required to address silent failures in seed data initialization, excessive method complexity in fillReadBundle, and inefficient list streaming.

1. ⚠️ Edge Case: initSeedDataFromResources now silently continues on failure
   Files: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1231-1241

   The try/catch added around `initializeEntity(entity)` means that if a critical seed entity (e.g. a default policy, role, or bot) fails to initialize, the system will start up with missing seed data rather than failing fast. The previous behavior propagated the exception, halting seed loading and surfacing the problem immediately.
   
   While resilience against one bad seed entity is useful, silently continuing could leave the system in a subtly broken state (e.g. missing the `ingestion-bot` or a default admin role). Consider distinguishing between optional and required seed entities, or at minimum collecting all failures and throwing at the end if any occurred.

   Suggested fix:
   // After the loop, fail if any critical seed entity was skipped:
   List<String> failures = new ArrayList<>();
   for (T entity : entities) {
     try {
       initializeEntity(entity);
     } catch (Exception e) {
       LOG.warn("Failed to init {} '{}'", entityType, entity.getFullyQualifiedName(), e);
       failures.add(entity.getFullyQualifiedName());
     }
   }
   if (!failures.isEmpty()) {
     throw new IOException("Failed to initialize seed entities: " + failures);
   }

2. 💡 Quality: fillReadBundle has 6 params and 3 boolean tracking vars
   Files: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:1655-1669

   The `fillReadBundle` method (lines 1655-1813) is ~160 lines long with 5 parameters, 3 boolean state-tracking variables, and multiple nested branches. Per the project's complexity guidelines (max 15 lines/method, max 5 params), this method would benefit from being decomposed. For example, the cache-hit path (lines 1665-1684), the DB-fetch path (lines 1689-1724), and the cache-populate path (lines 1756-1765) are each self-contained and could be extracted.

3. 💡 Performance: insertVersionExtensionsWithMetadata streams list 6 times
   Files: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:2597-2611

   In `insertVersionExtensionsWithMetadata` (lines 2597-2618), the `versionExtensionRecords` list is streamed 6 separate times to extract each field into its own list. For large batch imports this is wasteful. A single loop collecting all fields simultaneously would be more efficient and more readable.

   Suggested fix:
   List<UUID> ids = new ArrayList<>(versionExtensionRecords.size());
   List<String> exts = new ArrayList<>(versionExtensionRecords.size());
   // ... etc for each field
   for (var r : versionExtensionRecords) {
     ids.add(r.getId());
     exts.add(r.getExtension());
     jsons.add(r.getJson());
     versions.add(r.getVersionNum());
     keys.add(r.getChangedFieldKeys());
   }
   daoCollection.entityExtensionDAO()
     .insertVersionExtensions(ids, exts, entityType, jsons, versions, keys);

4. 💡 Quality: extractFqn line exceeds typical length limit
   Files: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:3262

   Line 3262 (`return node.hasNonNull("fullyQualifiedName") ? node.get("fullyQualifiedName").asText() : null;`) contains the magic string `"fullyQualifiedName"` repeated twice. Per the 'no magic strings' guideline, this should use the existing `FIELD_FULLY_QUALIFIED_NAME` constant (or equivalent) from the Entity class.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

Copilot AI review requested due to automatic review settings April 29, 2026 15:05
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

}

if (actual instanceof List<?> list) {
return list.stream().anyMatch(item -> expectedValue.equals(item.toString()));
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

matchCustomProperty(...) can throw an NPE when the custom property value is a list that contains a null element (it does item.toString() inside anyMatch). Since the helper is documented as “handles all null/missing cases gracefully”, guard against null list items (and/or handle NullNode -> null) so the function returns false instead of failing evaluation.

Suggested change
return list.stream().anyMatch(item -> expectedValue.equals(item.toString()));
return list.stream().anyMatch(item -> item != null && expectedValue.equals(item.toString()));

Copilot uses AI. Check for mistakes.
Comment on lines +199 to +205
if (entity == null || entity.getExtension() == null) {
cachedCustomProperties = Collections.emptyMap();
return cachedCustomProperties;
}
try {
@SuppressWarnings("unchecked")
Map<String, Object> props = JsonUtils.convertValue(entity.getExtension(), Map.class);
Copy link

Copilot AI Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getCustomProperties() caches Collections.emptyMap() as soon as entity.getExtension() is null. When this ResourceContext is constructed with a pre-loaded entity (constructor (resource, entity, repository)), resolveEntity() will not fetch missing fields, so an entity that supports extensions but was loaded without them will permanently appear to have no custom properties for the rest of the evaluation (because the empty map is cached). Consider lazily fetching the extension when missing (e.g., via entityRepository.getExtension(entity) / a lightweight reload with FIELD_EXTENSION) before caching the empty result, and only cache empty after that attempt.

Suggested change
if (entity == null || entity.getExtension() == null) {
cachedCustomProperties = Collections.emptyMap();
return cachedCustomProperties;
}
try {
@SuppressWarnings("unchecked")
Map<String, Object> props = JsonUtils.convertValue(entity.getExtension(), Map.class);
if (entity == null) {
cachedCustomProperties = Collections.emptyMap();
return cachedCustomProperties;
}
Object extension = entity.getExtension();
if (extension == null) {
try {
extension = entityRepository.getExtension(entity);
} catch (Exception e) {
LOG.warn(
"Failed to fetch custom properties for entity {}: {}",
entity.getId(),
e.getMessage());
}
}
if (extension == null) {
cachedCustomProperties = Collections.emptyMap();
return cachedCustomProperties;
}
try {
@SuppressWarnings("unchecked")
Map<String, Object> props = JsonUtils.convertValue(extension, Map.class);

Copilot uses AI. Check for mistakes.
@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Support for Custom Properties in Policy Rule Conditions

3 participants